Milestone 2: Model Validation

tl;dr

This notebook determines the model framework utilized for this PRD. It utilizes the results from the hyperparameter tuning step, and trains the optimal model for each algorithm on the v67 dataset. The resultant predictions of these optimal models are compared to a v68 validation dataset. The framework used to generate the highest performant model (e.g., features, hyperparameters) will be used in online model training for the subsetting of Beta in production use.

Features

Feature sets:

1. Only performance metrics.
2. Performance metrics and the highest found by Boruta with original dataset.
3. Performance metrics and the highest found by Boruta with equal labels dataset.
4. Performance metrics and all covariates.
5. Only the highest covariates found by Boruta (excluding performance).
6. Only the highest covariates found by Boruta (equal labels, excluding performance).
7. Utilize all covariates (excluding performance).

Validation Bootstraps

30 bootstrap replicates of v68 Release will be utilized to validate the models. There are 257697 profiles utilized for this purpose.

[1] "Loading previously created bootstraps from file"

Oversampling

The entire v67 dataset will be utilized for training. However, most of the matching algorithms are more performant with higher ratios of Beta to Release. Therefore, various levels of oversampled datasets are created for training. There are 70345 profiles utilized for this purpose.

[1] "Loading previously created oversampling training sets from file"

Model Training

Two matching methods were ultimately tested, as CEM and subclassing were producing extremely poor results.

• nearest-neighbors
• genetic matching

For all models the following diagnostics are reported:

• mean, median score against the 30 validation replicates
• Number of matched Beta samples
• QQ and ridge plots of original and matched Beta samples

Nearest-Neighbors, Malahanobis

Hyperparameters: hyperparameter_tuning_nn_malahanobis.Rmd

Feature selection and hyperparameter tuning for the nearest-neighbors models were all trained using the MatchIt library. These models are trained using the full, relevantly oversampled, training dataset.

FS 1

• caliper = 0.20
• calcloset = TRUE
• ratio = 2
• replace = FALSE
• 4x oversampling

Resultant scores: 8.201553210^{-4}, 8.12712610^{-4}

• Number of matched Beta training samples: 29814
• Number of matched Beta validation samples: 20922

FS 3

• caliper = 3.00
• calcloset = TRUE
• ratio = 1
• replace = TRUE
• 16x oversampling

Resultant scores: 0.0022122, 0.0022092

• Number of matched Beta training samples: 3259
• Number of matched Beta validation samples: 2199

FS 5

• caliper = 0
• calcloset = TRUE
• ratio = 1
• replace = FALSE
• 8x oversampling

Resultant scores: 0.0015801, 0.0015768

• Number of matched Beta training samples: 6208
• Number of matched Beta validation samples: 4226

Nearest-Neighbors Logit, Linear

Hyperparameters: hyperparameter_tuning_nn_logit_linear.Rmd

FS 1

• caliper = 0.4
• calcloset = TRUE
• ratio = 3
• replace = FALSE
• 4x oversampling
• Interactions across covariates

Resultant scores: 7.762891510^{-4}, 7.683315510^{-4}

• Number of matched Beta training samples: 44721
• Number of matched Beta validation samples: 30644

FS 5

• caliper = 0
• calcloset = FALSE
• ratio = 1
• replace = FALSE
• 8x oversampling

Resultant scores: 0.0013756, 0.001376

• Number of matched Beta training samples: 7453
• Number of matched Beta validation samples: 4825

Nearest-Neighbors GAM, Logit

Hyperparameters: hyperparameter_tuning_nn_gam_logit.Rmd

FS 1

• caliper = 0.5
• calcloset = FALSE
• ratio = 3
• replace = FALSE
• 4x oversampling
• Interactions across covariates

Resultant scores: 0.0011714, 0.0011762

• Number of matched Beta training samples: 44721
• Number of matched Beta validation samples: 30070

FS 5

• caliper = 0
• calcloset = FALSE
• ratio = 1
• replace = FALSE
• 4x oversampling

Resultant scores: 0.002441, 0.0024395

• Number of matched Beta training samples: 14907
• Number of matched Beta validation samples: 9704

Nearest-Neighbors Probit, Linear

Hyperparameters: hyperparameter_tuning_nn_probit_linear.Rmd

FS 1

• caliper = 0.4
• calcloset = FALSE
• ratio = 3
• replace = FALSE
• 4x oversampling
• Interactions across covariates

Resultant scores: 7.430882310^{-4}, 7.367975710^{-4}

• Number of matched Beta training samples: 44721
• Number of matched Beta validation samples: 30607

FS 5

• caliper = 0
• calcloset = FALSE
• ratio = 1
• replace = FALSE
• 8x oversampling

Resultant scores: 0.0013792, 0.001382

• Number of matched Beta training samples: 7453
• Number of matched Beta validation samples: 4958

Genetic Matching

Due to the very long training times of genetic matching, feature selection and hyperparameter tuning were not performed using bootstrap sampes. The models were previously trained using the full training sample. The resultant match beta subsets were serialized and uploaded to GCP.

Trained: feature_selection_genmatch_linear.Rmd

FS 1

Resultant scores: 0.001084, 0.001084

• Number of matched Beta training samples: 25846
• Number of matched Beta validation samples: 17684

Pop.Size 500

Resultant scores: 9.916087510^{-4}, 9.923251710^{-4}

• Number of matched Beta training samples: 25930
• Number of matched Beta validation samples: 17731

FS 3

Resultant scores: 0.0020662, 0.0020654

• Number of matched Beta training samples: 22648
• Number of matched Beta validation samples: 15929

FS 5

Resultant scores: 0.0020716, 0.0020748

• Number of matched Beta training samples: 22469
• Number of matched Beta validation samples: 15759

Results

Nearest-Neighbors Probit, Linear produces the optimal matching results. It yields the lowest scores, without dropping large sample of the Beta profiles. Unsurprisingly, matching directly, and only, on the performance metrics yields the best results. Interestingly, it appears that interactions across covariates greatly help. Fortunately, this model architecture is relatively cheap from a computational perspective.

Balancing on additional covariates (e.g, FS3 or FS5), outside of the performance metrics, yields sub-optimal results. Nearest-Neighbors Probit, Linear again yields the lowest score (0.0013792, 0.001382); however, this is at the expense of a greatly reduced Beta samples size (4958). Genetic matching is able to retain a much larger sample size (15929), at the expense of greater error (0.0020662, 0.0020654).

Conclusion: Optimal model framework

• matching algorithm: Nearest-Neighbors Probit, Linear
• features: Only performance metrics
• hyperparameters:
  • caliper = 0.4
  • calcloset = FALSE
  • ratio = 3
  • replace = FALSE
  • 4x oversampling
  • Interactions across covariates

Push to GCP

The boostrap replicates for validation, oversampled traininig datasets, resultant quantile calculations, and matched datasets are saved to an R image. Then uploaded to the project GCP bucket.

Auto-refreshing stale OAuth token.

[1] "Previously trained results already exist: data/milestone2/validation_results_20191121.RData"